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ABSTRACT 

In a study designed to determine the impact of state 
curriculum initiatives on curriculum framework, student assessment, 
and textbook adoption in the science curriculum, researchers 
interviewed state education agency staff, completed four case studies 
in four districts, and surveyed 100 school districts at random in 
each of the three diverse states sampled. Following a detailed 
summary of document contents, this document uses the survey results 
to identify and describe the impacts of state initiatives on science 
curricula. The development, design and specificity, and dissemination 
of state curriculum frameworks are discussed and state assessment 
programs are examined in terms of content alignment, scoring metrics, 
sampling items, teaching to the test, item content, and investigatix« 
instruction. State textbook adoption policies are discussed in terms 
of curriculum and framework alignment, resources for textbook 
training and selection, and adoption incentives and five strategies 
related to curriculum hange implementation and technical assistance 
are described. State .litiatives are shown to have either neutral or 
positive impacts on science curricula. A final section discusses the 
implications of study results for state policy. (4 references) 
(CLA) 



********************************************************************* 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

************** ^ ************************** *^************************** 



CO 

© 
o 

00 
00 
Q 
H 



CCS 

working papers 



Published by the Education Commission of the States 



Designing State Curriculum Frameworks 
and Assessment Programs 
to Improve Instruction 



n« U.S. DEPARTMENT OF EDUCATION 

Offve of Educational Rewarch and Improvement 

EDUCATIONAL RESOURCES INFORMATION 

- CENTER (ERIC) 

V) This document hps been reproduced ae 

received from tha person or organisation 

originating it 

V Minor changes have been made to improve 
reproduction quality 



e Pomfs of view or opinions stated in this docu 
menf do not necessenty represent official 
OtRI position or policy 



BEST COPY AV/tRABU 



IS 

ERIC 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)/' 



Designing State Curriculum Frameworks 
and Assessment Programs 
to Improve Instruction 



By 

Jane Armstrong, Education Commission of the States 
Alan Davis, RMC Research Corporation 
Allen Odden, University of Southern California 
James Gallagher, Michigan State University 

April 1989 

EDUCATION COMMISSION OF THE STATES 
1860 Uncoin Street, Suite 300 
Denver, Colorado 80295 



This report Is based on a study by the Education Commission of the 
States funded by the National Science Foundation (Grant Number 865- 
0060). The views expressed here do not necessarily represent those of 
the National Science Foundation. 



Copies of this book are available for $6.50 from the ECS Distribution 
Center, Suite 300, 1860 Lincoln Street, Denver, Colorado 80295, 303-830- 
3692. Ask for No. SI-89-5. 

© Copyright 1988 by the Education Commission of the States. All rights 
reserved. 

The Education Commission of the States is a nonprofit, nationwide 
Interstate compact formed In 1965. The primary purpose of the 
commission Is to help governors, state legislators, state education officials 
and others develop policies to Improve the quality of education at all 
levels. Forty-eight states, the District of Columbia, American Samoa, 
Puerto Rico and the Virgin Islands are members. The ECS central offices 
are at 1860 Lincoln Street, Suite 300, Denver, Colorado 80295. The 
Washington office is In the Hall of the States, 444 North Capitol Street, 
Suite 248, Washington, D.C. 20001. 

It Is the policy of the bducatlon Commission of the States to take 
affirmative action to prevent discrimination in Its policies, programs and 
employment practices. 



ii 4 



-TABLE OF CONTENTS 

INTRODUCTION 1 

IMPACTS OF STATE INITIATIVES 3 

DESIGN OF STATE INITIATIVES 5 

State Curriculum Frameworks 5 

State Assessment Programs 10 

State Textbook Adoption Policies 15 

Implementation Strategies and Technical Assistance 16 

IMPLICATIONS FOR STATE POLICY , 21 



ERIC 



iii 

5 



SUMMARY. 



Interest In science education Is mounting In the United States and state 
governments are moving to Influence curriculum content, selection of 
instructional materials, assessment of student achievement and other 
policies central to the Instructional process. Because of this, the National 
Science Foundation (NSF) funded the Education Commission of the States 
(ECS) to study the effect of state policies on the science curriculum. 
ECS studied the design and use of state policies related to curriculum 
frameworks, student assessment and textbook adoption; state 
Implementation strategies; and the local conditions that facilitated 
Implementation. It also analyzed the implications for state policy. 

ECS looked at the Impacts of these policies in three states chosen to 
represent varied mixes of policies and a continuum of state control of 
education versus local control. Researchers Interviewed personnel in 
each state's department of education and f Air of its school districts. 
These Interviews became the basis of case studies and cross-site analyses. 
A survey based on case-study findings subsequently was sent to a 
random sample of curriculum supervisors in 100 districts in each state. 



Impacts of State Initiatives 

In the three states studied, there was little evidence that state science 
activities produced negative Impacts. Impacts instead ranged from 
neutral to positive, and neutral Impact was frequently, though not always, 
found In districts that already had science programs of unusually high 
quality. Though the researchers expected to hear concern about state 
control and negative reactions to testing, they found little evidence of 
either. 

• In many districts, state initiatives produced greater local emphasis 
on science. 

• Many districts used state science frameworks to revise their own 
curricula and to train teachers. 

• State Involvement either improved the quality of science materials 
or produced no change. 

• State activities strengthened administrative ties between schools and 
districts. 

• State initiatives significantly increased hands-on instruction. 

• State testing encouraged restricts to emphasize science but did not 
result In "teaching to the test." 




Design of initiatives 



A central finding of the study was that using curriculum frameworks, 
assess-ment, textbook selection and technical assistance in a highly 
coordinated and conceptually coherent fashion powerfully affects the 
quality of instruc-tion. Though the design of particular initiatives In the 
three states varied significantly, the experience of all three supported that 
central finding. 



Variations in Curriculum Frameworks 
Impacts were related to: 

• Who was involved in the development of the frameworks 

• How many people were involved 

• How national resources or expertise was tapped 

• How successfully the framework was disseminated 



Variations in Assessment 

All three states administered multiple-choice tests to students in selected 
grades. Where two of the states used their curriculum frameworks to 
guide test construction, the third state instead continued to use a 
commercially available norm-referenced test less well-fitted to the 
curriculum. Impacts were related to: 

• How well the test content was aligned with the curriculum 
framework 

• Whether the state assessment referenced norming samples or a 
mastery criterion 

• The opportunity to teach to the test 



Variations in Textbook Selection 

Though one state allowed districts to choose their own textbooks, the 
other two states centralized the textbook adoption process. Impacts 
were related to: 

• How well the textbook selection criteria supported the curriculum 
framework 

• Available resources and training provided for the selection process 
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• The provision of financial incentives for districts to select from the 
state adoption list 



Variations in Implementation 

Implementation was facilitated in the three states when: 

• State curriculum frameworks were widely disseminated to school 
and district personnel 

• Educators throughout the state were involved in the development 
of the curriculum frameworks or regional conferences related to 
their use 

• States used regional service centers to provide technical assistance 
to districts to promote Implementation over a period of time 

• Title II funds were used for teacher training 

• Implementation was coordinated with other school Improvement or 
accountability efforts 



Local Conditions That Facilitated Impacts 

Where the Impact of state policies to Improve the science curriculum 
was high, the following conditions tended to prevail. 

• An accountability pressure resulting In a district's or school's strong 
desire to do well on a state assessment 

• District leadership and commitment to teaching science 

• A match between a state framework and district philosophy or 
willingness by a district to "buy In" to the slate philosophy 

• Centralization at the district level of science curriculum and 
Instructional materials 

• Teacher involvement in developing curricula to support the state 
framework 

• Availability of textbooks, hands-on materials, activity kits 

• The presence of a district science supervisor or of lead teachers 

• Training and assistance of teachers 
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• Monitoring of implementation by district and school leaders 

Among the districts where the Impact of state policy was low - where 
state Initiatives caused few changes in curriculum and Instruction - were 
those thot already had very strong science programs, rejected the 
philosophy of the state framework or were focusing on other priorities. 



Implications for State Policy 

General Strategies 

• Impacts are greatest when states integrate curriculum frameworks, 
student assessment and textbook selection, with the framework at 
the core. Curriculum frameworks should be developed first, and 
revision cycles should begin with the curriculum. 

• Direct, regular communication between curriculum units arid assess- 
ment units Is essential. These units should be housed close to each 
other and placed under a single upper-level administrator. 

• Some of the subject-matter experts who develop curriculum frame- 
works should also develop assessments and criteria for textbook 
selection. 



Curriculum Frameworks 

• Frameworks should reflect the thinking of leading national 
organizations such as the National Science Foundation, American 
Association for the Advancement of Science and National Science 
Teachers Association. 

• Though the participation of influential districts In the development of 
frameworks is Important, quality should not be compromised to 
achieve broader participation. 

• Objectives must be broad enough to channel instruction, not so 
specific that Instruction becomes atomistic. 

• Adherence to frameworks should be optional. 



Assessment 

» Assessment and state science objectives should be aligned so that 
Impacts are greater and more positive. 

• Test results should be reported by schools to provide an 
appropriate level of accountability. 
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• To avoid "teaching to the test," test items should not be repeated 
year after year and tests should be kept secure. 

• Performance testing may promote activity-based instruction more 
effectively than short-answer formats. 



Textbook Selection 

• States and national organizations should together develop a core 
curriculum in science. 

• Texibooks should reflect the current goals of science, such as 
everyday applications, technology and society, the process of 
science and ethical issues. 

• Criteria for textbooks should reflect the state curriculum framework. 

• Sufficient resources should be allocated to textbook selection. 

• Districts should have financial incentives to purchase state-adopted 
textbooks, which fosters curriculum/state framework alignment. 



Dissemination, Implementation, Monitoring 

• Strategies for disseminating the state science framework should be 
comprehensive. 

• Districts should receive discretionary resources to aid implementation. 

• States should provide districts with technical assistance in teacher 
training and curriculum Implementation. 

• States should sponsor regional centers that aid implementation. 

• Monitoring should be done through regulations or incentives to 
promote implementation. 



Content 
States should: 

• Establish broad goals for science education. 

• Identify a "core" of science knowledge that should be taught to all 
students. 
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• Continue to support the development of new curriculum materials. 

• Develop a national directory of exemplary science programs. 

Professional Community 

The professional education community should: 

• Support professional activities and leadership development in the 
science-education community. 

• Develop and support a cadre of district curriculum specialists and 
lead teachers who can serve as resources in large school districts. 

• Fund the development of the component of the National Board for 
Professional Teaching Standards that assesses science teachers. 

• Help make science a high priority by linking it to other formal and 
Informal science activities. 
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INTRODUCTION. 



Few policy makers understand how much effect the design of state 
curriculum frameworks and assessments have on improving curriculum in 
school districts. Because almost every state has an assessment program 
and some type of curriculum framework (also known as curriculum guides 
or objectives), state policy makers must realize the implications of their 
decisions regarding the development and design of related state 
Initiatives. This analysis for state policy makers presents different design 
options for curriculum frameworks and assessment programs and examines 
their impacts on school districts. 

information is based on an ECS study of state initiatives of curriculum 
frameworks, student assessment and textbook selection in three states. 
The states were selected to represent state ve^us local control and 
different designs of Initiatives and strategies for implementation (for 
anonymity, they will be referred to as the Western State, Midwestern 
State and Eastern State). Researchers interviewed staff in each state 
education agency to develop a description of the state initiatives and 
their intended use by school district personnel. 

Case studies were completed in four diverse school districts in each state 
to understand how the state initiatives were being used, their impacts on 
improving curriculum and the local conditions that facilitated any impact. 
One hundred school districts in each state then were randomly surveyed 
to see If the impacts found in the four districts were prevalent 
throughout the state. 

Science curriculum was chosen as the focus of this study because of its 
interest to the funding agency, the National Science Foundation; the 
considerable attention it has received as the cornerstone of the nation's 
technological and economic competitiveness; and the urgent need to 
improve the dismal achievement scores of U.S. students on national and 
international science comparisons. 

Science also has a unique place in the K-8 curriculum. Historically, it has 
taken a back seat to reading and mathematics and is taught by 
teachers who are frequently not highly trained In science or supported 
by district expertise. Yet, science is considered by most educates to be 
an important part of the curriculum. This paradox makes the science 
curriculum an ideal subject for study because state policies and initiatives 
have a more visible impact in subject areas in which districts cr,d 
teachers lack expertise and have a strong self-interest in using stale 
resources to improve their curriculum. 
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JMPACTS OF STATE INITIATIVES. 



In an era of state reform and an increasingly prominent role In Improving 
education, the study fount* that state Initiatives had either neutral or 
positive Impacts. In dlstricrs where the state Influence v>as neutral, there 
frequently were existing programs of unusually high quality and the state 
Initiatives did not prompt Improvements. Although researches expected 
to hear concern about Increasing state control and negative reactions to 
testing or to the Imposition of a state curriculum, there was little 
evidence to support this. The Impacts found were: 

• In many districts, state Initiatives produced greater local emphasis 
on science. 

No district reported a decrease in elementary science instruction. 
In the Western State, 69% of survey respondents (directors of 
Instruction) reported Increases and attributed them to state action; 
In the Midwestern State, 39% reported Increases, and, In the Eastern 
State, 23%. None of the three states mandated minimum minutes 
of science Instruction, and none lengthened the school day; 
Increases In science Instruction resulted from decreases In other 
areas of Instruction. 



• Many districts used state science frameworks to revise their own 
curricula and to train teachers. 



More than one-third of the districts surveyed changed their science 
curriculum as a result of state frameworks; more than 85% of the 
districts that made changes reported that they provided training to 
help teachers make the changes. In the Eastern and Midwestern 
States, districts with fewer than 3,200 students were less than half as 
likely to have changed the curriculum as districts with more than 
15,000 students. Only one district made no effort to change its 
own curriculum because It objected to the philosophy of the state 
framework. 

State Involvement either Improved the quality of science materials 
or produced no change. 

Seventy-three percont of districts in the Western State associated an 
Improvement in instructional materials with state Initiatives. The 
percentage reporting improvement was smaller In the Midwestern 
State (39%) and Eastern State (34%), but no respondent reported a 
decline In quality. 

State activities strengthened administrative ties between schools and 
districts. 
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In all districts where the impact of state activity was high, staff 
development was centrally coordinated. Science curriculum 
specialists generally found themselves empowered by the new 
importance attached to science. Although the greater influence of 
state depart-ments of education and science specialists limited the 
discretion of teachers to some extent, the teachers interviewed 
generally supported the need for a common basis for staff 
development, materials acquisition and instruction. 

State Initiatives significantly Increased hands-on Instruction. 

Because classroom Investigations and development of the "process 
skills'' of observing, measuring and using evidence were curriculum 
objectives in all three states, the teaching of science through 
inquiry increased dramatically. In many cases, though, the hands- 
on activities described by teachers were their own demonstrations 
rather than student-led inquiry. 

State testing encouraged districts to emphasize science but did not 
result In "teaching to the test." 

Testing appeared to affect the broader features of instruction rather 
than the specifics. Teachers described state science tests as a 
"placeholder for science," for example, and few of them Indicated 
familiarity with specific test items. 



DESIGN OF STATE INITIATIVES 



The three states selected for this study represented a range of 
approaches that states can employ to Improve the science literacy and 
preparation of students. Four aspects of state science initiatives were 
analyzed: design of curriculum frameworks, science assessment, textbook 
selection and technical assistance. The study found that a state effort 
that employs all four of these elements In a highly coordinated and 
conceptually coherent fashion Is a powerful tool to affect the quality of 
Instruction In schools and districts. Because of its power, such a 
state-directed approach has the potential for significant harm as well. 
While the Western State, for example, appeared to be using directive 
state leadership in curriculum to bring about substantial improvements in 
science education, there can be no assurance that this approach would 
not be harmful if used in a different state context. 



State Curriculum Frameworks 

Forty-seven states have developed or adapted frameworks that range 
from very general recommendations to well-articulated and 
comprehensive guide-lines that integrate curriculum development, 
textbook adoption and teacher training. The purposes of these 
frameworks are diverse. Initially, states developed them to provide a 
basis for uniform instruction and to define some minimal level of 
curriculum content, quality and appropriateness. 

Frameworks have also served as a useful model for small districts which 
do not have the resources to develop their own. State frameworks help 
districts plan or revise their curricula with a philosophy of instruction that 
is consistent with the state's intent. Finally, with the opportunity to align 
their curriculum with the state assessment, many states have developed 
frameworks to reflect state-of-the-art trends and "lead" the curriculum, 
usually emphasizing higher-order skills and assessing a range of skills. 

Four events or characteristics of curriculum frameworks had a bearing on 
their use and impact in the school districts studied. These were develop- 
ment, design, specificity and strategies for dissemination. 



Curriculum Framework Development 

Two of the states in this study involved many state educators to develop 
the frameworks. The third state used a more select group of state 
science educators as well as national experts. Although the involvement 
of many state educators encouraged "buy-in" to the state framework 
and aided dissemination, this process also resulted in more of a 
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"consensus* document, one that reflected current practice rather than 
cutting-edge trends. 

The Midwestern State's curriculum framework for science education was 
developed cooperatively by the state department of education, state 
affiliate of the National Science Teachers Association (NSTA), local school 
district personnel and representatives of higher education Institutions. A 
17-member committee reviewed curricular materials from California, 
Colorado, North Carolina, the National Assessment of Educational Progress 
(NAEP) and NSTA. Objectives were developed through meetings over a 
two-year period, and 600 teachers and supervisors throughout the state 
reviewed draft copies. Most educators viewed the curriculum framework 
as of high quality and consistent with current trends in science 
education. 

In the Eastern State, the curriculum framework was developed at the 
same time as standards in other curriculum areas. Although the state 
science supervisor led the development of the science standards, the 
assistant superintendent for curriculum and instruction coordinated 
development of all subject-area standards. Certain guidelines were 
suggested for the format of all the standards, which affected both their 
design and content. One guideline, for example, suggested that 16 
objectives be developed for each grade and subject area, regardless of 
the appropriateness for the subject area. 

Almost 100 teachers and curriculum specialists were invited from across 
the Eastern state to develop the "learner outcomes" (specific content 
objectives) for science. These committees were asked to review school 
district curricula, guides from other states and the earlier basic skills 
objectives and to recommend outcomes for each grade. The result was 
a set of learner out-comes that reflected current practice in the state. 
Under the leadership of the state science supervisor, another section was 
included. Called "program goals and objectives," this section articulated 
the philosophy, process and concepts of science which were meant to 
serve as the unifying themes for the learner outcomes. 

The Western State's science framework was a revision of an earlier 
document and was developed by a curriculum committee of 15 science 
educators. Members were the state's leading experts in science 
education, including a mixture of classroom teachers, district and county 
office curriculum coordinators and university professors. The committee's 
work was coordinated by a state department of education team 
composed of consul-tants and the state science supervisor. This team 
articulated the new curriculum agenda and brought in national 
consultants to interact with the committee. Because a limited number of 
educators were involved in developing the framework, the result was a 
state-of-the-art document designed to encourage districts to develop or 
revise their curricula in light of current thinking in science education. 
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Design and Specificity of the Framework 



The design of each state curriculum framework affected its use. One 
issue was the level of specificity and coherence of the framework. Most 
state frameworks describe the philosophy of science education and 
highlighted the major concepts, processes, content and related issues. 
The goal was to give curriculum specialists and teachers guidance in the 
intended and appropria+e uses of the framework. 

The Western State used two documents to describe its science 
curriculum. The framework included expectations for student 
achievement in biological science; earth science; physical science; 
science, technology and society; and ©rnical issues. These areas were 
considered the "knowledge" of science and related to another dimension, 
the "processes" (observing, communicat-ing, comparing, organizing, 
relating, inferring and applying). For each "knowledge" objective (which 
included concepts, ethical concerns and technological applications), 
corresponding "thinking-process" skills were presented. The second 
document, a model curriculum guide, supplements the framework and 
suggested a learning sequence, delineating concepts, skills and activities 
appropriate for learners In grades K-8. 

Teachers in the Western State reacted positively to the model curriculum 
guide, saying content and model lessons helped them lead discussions, 
frame questions and design activities that contain multiple levels of 
learning. 

Overall, the design of the Western State's framework was a coherent 
strategy for teaching and learning. Knowledge and concepts were 
supported by process skills and teaching strategies. Neither the 
framework nor tho model curriculum guide was mandatory. They did, 
however, provide guidance to teachers and creative strategies to teach 
science. Teachers with a strong science background could build on this 
Information. Teachers less sure of their science skills were able to 
develop an integrated, hands-on curriculum with guidance from the 
framework, curriculum guide and staff development activities. 

The Midwestern State's science curriculum framework approached science 
Instruction by defining a philosophy, a set of broad goals and specific 
objectives for each goal. The philosophy defined the nature of science 
and described its relationship with society, the learner and the school 
curriculum. Broad statements of desired outcomes were organized into 
seven goals: life science; physical science; earth/space science; science 
process; science, technology and society; science attitudes; and the 
nature of science. Each major goal had several subgoals (for example, 
life science has subgoals of systematics, cellular and molecular biology, 
heredity, evolution, etc.). Within each subgoal were 5-10 objectives, such 
as "living things have adaptations which enable them to survive." 

The Midwestern State suggested the framework was designed to assist 
administrators and teachers in planning, developing and implementing K-9 
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science programs and to provide some guides for grades 10-12 
instruction. The state expected districts to compare their existing curricula 
with the state framework and make adjustments if necessary. 

The framework had seven goals and subgoals which covered the major 
con-tent of science, science processes, the nature of science and 
societal concerns. The framework did not make the interconnections of 
content and process found In the Western State's framework, nor did it 
suggest a series of teaching strategies as found in a model curriculum 
guide. Although districts found It to be a straightforward process to 
compare their existing curriculum to the state objectives, they were not 
provided with additional teaching strategies, links between science 
content and process or to other subject areas to facilitate instruction. 

The Eastern State's science curriculum framework, developed by almost 
100 state educators and administrators, was organized into four parts: 
program goals, objectives, guidelines and learner outcomes. Program 
goals and objectives had their foundation in the state's quality standards 
for schools and NSTA's "Position Statement for School Science." 

The framework was organized by four program goals exemplified by 
program objectives. For example, objectives under the one goal, the 
nature of science, included process skills, conceptual themes, career 
Information and the developmental appropriateness of the objectives. 
Program objectives were further defined by guidelines. One guideline 
describing a conceptual theme was: "Provide experiences in which 
students investigate the interrelationships which exist among living and 
non-living things and their environments." 

The second part of the Eastern State's framework presented about 15 
learner outcomes for each grade, K-12. With such a small number of 
outcomes, these were intended to be illustrative rather than 
comprehensive. 

The illustrative nature of the learner outcomes, coupled with the state 
accreditation requirement that teachers must document when they 
taught each outcome, led to a diversity in science program quality. The 
15 or so illustrative outcomes for each grade were not intended to 
provide a "conceptual whole" of what science should be at that grade. 
School districts or teachers were expected to embed these outcomes in 
their science units or lessons. In districts where there was leadership and 
expertise in science education, the state curriculum framework was 
integrated into a thoughtful, well-articulated and frequently integrated 
science program. However, in small districts where curricular leadership 
in science was limited, teaching of the state objectives was left to the 
discretion of individual teachers, often resulting in a discrepancy between 
the state's intent and actual implementation. 

The accreditation requirement also focused teachers' attention on the 
learner outcomes rather than the program goals and guidelines - the 
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most instructlonally coherent and comprehensive part of the framework. 
Thus, the framework design resulted in different Impacts in two types of 
districts. Where the district had expertise or access to expertise, the 
framework had a positive Impact on the content of science curricula. In 
other districts, where science was not a priority, the state framework had 
very little impact on improving science instruction because it was not 
readily usable by educators unfamiliar with science education. 



Dissemination of State Frameworks 
to Local Educators 

Strategies for disseminating state frameworks varied greatly among the 
three states studied. Both the Midwestern and Eastern States involved 
many state educators during the development phase to promote buy-in 
and dissemina-tion of the state frameworks. The Western State promoted 
understanding and implementation of the framework by sponsoring a 
series of regional workshops, developing a three-year cycle of 
Implementation and providing ongoing technical assistance through 
regional centers. 

The Midwestern State sponsored about 15 regional workshops to promote 
use of the state framework. Limited resources prohibited dissemination of 
the framework to every school. Although a copy was sent to the 
superintendent of every school district, many teachers did not see or 
receive a copy of the state objectives prior to the administration of the 
state assessment. 

The Eastern State was more successful in disseminating its framework to 
school districts and teachers. First, the science standards were part of a 
larger package of state objectives In several learning areas. Second, 
the framework grew out of quality standards for schools, and the 
accreditation standards required teachers to document use of the learner 
outcomes. Third, the involvement of almost 100 educators from half of 
the state's school districts helped communicate the nature of the 
framework to most of the districts. Finally, the state department of 
education sponsored a series of regional workshops to explain the 
framework to local educators. Uniquely, the state also worked with the 
institutes of higher education to provide a course in the use of the 
science framework as part of undergraduate teacher training. 

Although the Western State involved the fewest number of educators in 
the development of the science framework, it was, perhaps, the most 
successful state in disseminating the framework. The small size of the 
development committee and the coordinating efforts of the state 
department of educa-tion staff allowed a more directive approach to 
the development process. The result was a state-of-the-art, 
comprehensive and highly usable set of documents to help districts 
improve science instruction. 
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Because the curriculum frameworks were at the heart of the state's 
educa-tional reforms, the state provided resources for regional workshops 
and technical assistance through Its regional centers, Although the 
curriculum framework and guides were recommended rather than 
required, districts chose to use them because they were of such high 
quality and were the basis for the mandated state testing and the 
textbook selection process. 



State Assessment Programs 

Statewide assessments of science achievement were important elements 
of efforts to improve science education in each of the three states 
studied. In each case, pencll-and-paper tests with multiple-choice 
formats were administered to students in selected grades. Beyond this 
fundamental similarity, differences in the design and implementation of 
the assessments were associated with how they were perceived and how 
they affected local science education. These differences fell into four 
categories: content alignment, scoring metric, item sampling and 
implications of item content for inquiry activities. 



Content Alignment 

Content alignment refers to the match between the items making up the 
assessment and the curriculum framework developed by the state. The 
Western and Midwestern States developed tests to reflect their curriculum 
frameworks, while the Eastern State purchased a commr rcially published 
standardized test and later developed a curriculum framework 
Independently. The contrast between these approaches resulted in 
strong differences in the perceived content validity of the tests in the 
schools visited. 

In the Western and Midwestern States, the development of the curriculum 
frameworks preceded test development and the frameworks provided the 
content map that subsequently guided test construction, The 
Midwestern State sampled 30 to 32 objectives at each of the tested 
grades from the curriculum framework. Three items were constructed to 
measure each one. In the Western State, a much larger item pool was 
developed, similarly derived from the curriculum. 

The approach in these two states contrasted with that employed in the 
Eastern State. That state adopted a nationally normed, commercially 
published test. The science framework was subsequently developed, 
without directly referencing the test. The state department of education 
reportedly examined the match between test items and state science 
objectives and found that a large proportion (more than 75%) of the 
items corresponded to state objectives. 
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However, teachers and science curriculum coordinators throughout the 
Eastern State reported that the test did not match either the state or 
local objectives and was therefore not useful to them, When students 
did poorly on the test, teachers attributed the low scores to the lack of 
fit between assessment and instruction. In the absence of any direct 
pressure to raise science scores, teachers did not report changing their 
Instructional practices on the basis of test results. 

In the Western and Midwestern States, the fit between test items and 
state curriculum was perceived to be high. As a consequence, districts 
thai undertook to Implement the state curriculum frameworks were more 
likely to view the test as a valid indication of their success. Central 
office administrators in half of the eight districts studied in these two 
states planned to use the recent science scores cs a baseline for 
charting science improvement. Several teachers voiced similar intentions. 
For these districts, the test served in part as a motivator, a means of 
keeping score. On the other hand, the test was dismissed as invalid by 
the science coordinator in one district in the Midwestern State where the 
local curriculum was significantly different from the state guidelines. 



Scoring Metric 

Among the three states studied, the Western and Eastern States 
referenced norming samples in reporting scores, while the Midwestern 
State referenced a mastery criterion. In the Western State, results from 
the first-year assessment were used as the baseline or reference point. 
A mastery criterion used in the Midwestern state was a predetermined 
guideline that required students to answer correctly at least two of the 
questions on each objective to achieve mastery. The criterion-referenced 
scoring produced extremely low scores and came as a significant shock 
to most districts, including high-income districts. Norm-referenced scoring, 
particularly when based on state norms as in the Western State, 
produced fewer surprises. 

The Midwestern State set out mastery criteria for its science assessment 
based on tradition and professional judgment rather than empirical 
methods. Each objective was measured by three items; two out of 
three correct indicated mastery. Scores were classified into four groups 
according to the percent of objectives mastered, with 75% required for 
the top, or mastery, category. 

In a state accustomed to having well over 80% of students showing 
mastery of reading, mathematics and language arts in the 4th grade, it 
came as a shock that only 39% of students qualified in science on the 
recent assessment. Mastery percentages in grades 7 and 10 were still 
lower. Several superintendents protested the test, fueled by the discovery 
that three items had been miskeyed. Several districts began 
self-examinations which led to efforts to improve science. In the end, 
the state department of education chose to defuse the situation by 
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de-emphasizing the importance of the science test and by making 
participation in the science assessment voluntary. (Almost a year later 
the state board reversed itself and made the test mandatory again.) 

The political impact of the Midwestern State's assessment was very 
directly associated with Its criterion-referenced scoring. Had scores been 
norm- referenced, a majority of districts likely would not have found 
themselves below an established point of reference, such as a national 
or state average. On the other hand, the power of the test to 
dramatize the need to improve science education would have been 
greatly reduced. The case emphasizes the need to conduct an 
empirical piloting of a criterion-referenced test to avoid setting a criterion 
too high to be politically tenable. 



Sampling Items and Teaching to the Test 

It is widely known thot pressures to increose scores con result in the 
teaching of particular test items rather than the broader objectives that 
underlie them. When this occurs, assessment can serve to restrict 
instruction rather \ian improve it. One partial solution to this problem is 
to vary the items on a test from year to year by creating multiple forms, 
writing new items or by sampl-ing from a large item pool. Among the 
three states studied, one created a single form of its science test, one 
employed two parallel forms, and one created multiple forms by 
sampling items from a large pool. 

The state curriculum frameworks in the three states did not provide long, 
specific lists of knowledge and skills to be taught. Instead, they provided 
lists of 15 to 100 objectives per grade, some of which were quite broad. 
For example, one objective for 6th-grade science in the Midwestern State 
was: The learner will gain understandings of the major structures found 
in the universe, including the sun and its planets." To test this objective 
in a multiple-choice format, it was necessary to narrow the focus. A 7th 
grader could demonstrate "mastery" of the objective by responding 
correctly that Jupiter was the largest planet, and that the universe was 
larger than a galaxy or a constellation. Obviously, the state intended for 
instruction to touch on far more than the content of these test items. 

The Midwestern State produced only one version of its science test, 
which was intended to be used annually. Only one teacher explicitly 
indicated that he intended to teach specific items of the test that his 
students had missed the previous year. Knowing that several principals 
and superintendents had been embarrassed by very low scores, it is 
likely that other teachers might have b^en motivated to become familiar 
with easily instructed test items and teach their content explicitly. Once 
this practice became widespread, the test would no longer serve as a 
valid measure of the general achievement of the broader objectives. 
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The Western State approached the problem of limited test items by 
creating 36 different forms of the 8th-grade science test. Each form 
contained 15 science questions, drawn from a pool of 640 Items. The 
same Items were used year after year, but test design discouraged 
attempts to teach to specific items. It would be unlikely that a teacher 
would collect all 36 forms and reconstruct the complete test. Even so, 
the test represented a much broader range of content than found in 
the Eastern State's standardized test (40 items on each of two forms) or 
the Midwestern State's test (96 Items). The use of multiple forms 
represented a positive solution to the problem of teaching to test items. 



Item Content and Investigative Instruction 

In each state, science frameworks included objectives dealing with the 
investigative processes of science. State science coordinators believed 
that these processes were under-emphasized in K-8 science instruction. 
They also agreed that these objectives were best approached through 
hands-on classroom activities rather than through direct lecture or 
reading. The assessment of these "science process" objectives posed 
significant questions: Can science process objectives be validly assessed 
In multiple-choice tests? Can multiple-choice assessments of science 
process objectives encourage hands-on Instruction? Answers to these 
questions were largely outside the scope of the present study, but some 
observations can be made within the limitations of the data. 

Assessments in each state included items designed to measure science 
pro-cess objectives in multiple-choice format. The Midwestern State's and 
Eastern State's tests were analyzed for content at the 7th- and 8th-grade 
levels to examine the tests' Instructional implications. (The Western State's 
Items were not available for analysis). Three questions were examined: 
(a) What proportion of questions measures knowledge that potentially 
could ba taught In association with classroom or outdoor investigations at 
school? (b) What proportion of questions asks a child to predict what 
will happen in an empirical trial, to design an experiment or to identify 
needed information to resolve a question? (c) What proportion of 
questions asks a child to arrive at a conclusion or Infer a rule from 
evidence provided? 

To examine the proportion of questions measuring knowledge that could 
potentially be taught through school investigations, Items were rejected if 
they were primarily definitional (e.g., "The material that makes up the cell 
wall of a plant is cellulose") or if their subject could not be examined or 
practiced first-hand by most students (e.g., "Which of the following is an 
effect of the moon on the earth?") For both tests, less than half the 
Items were judged to be conducive to instruction through investigation 
(see Table 1). 

Both tests contained questions that asked children to predict the 
outcome of an empirical trial (e.g., "Which of the bottles pictured will 
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produce the highest-pitched sound wr.jn you blow across the mouth?" 
"Which of the following mixtures will result In a solution?") These were 
included in the count only if students could conduct the trial themselves. 
These questions were combined to ask students to choose the best 
design for a study to test a hypothesis (e.g., "You want to determine if 
light-colored objects absorb more heat than dark-colored objects. Which 
investigation pictured below will solve this problem?") Such questions, 
combined, made up 12% of both the Midwestern and Eastern State's test 
Items. 

Table 1 

Percentage of Assessment Questions Conducive 
To Instruction By Investigation 



Question Type Percent of Items 

Midwestern State Eastern State 



Could Instruct by Investigation 45% 45% 

Predict Outcome or Design Trial 12% 12% 

Infer Rule From Evidence 17% 13% 



A third category of questions provided children with data, often in a 
table or graph or with a sequence of pictures, and asked them to infer 
a rule from the evidence. For example, one item displayed a line 
graph relating the amount of time water is heated with the temperature 
of the water. It asked: "From the information given in the graph, select 
a rule which describes the changes." Of the Midwestern State's test 
items, 17% were of this type; of the Eastern State's test, 13%. 

In addition, both tests contained items identified by the test designers as 
measuring science process, but which essentially measured definitional 
knowledge or concepts that could be readily conveyed in readings or 
lec-ture (e.g., "The seismograph is an instrument that detects vibration." 
"Most scientists consider scientific theories to be explanations that may be 
revised.") For this reason, the proportion of items that the test authors 
considered to measure science process did not appear to indicate 
whether a test would in fact promote investigative activities in instruction. 



From this cursory analysis, it would appear that both tests encouraged 
Instruction in the design and carrying out of investigations, reading tables 
and graphs and Inferring rules from patterns of numbers or events. One 
district in the Eastern State, which had an activity-based science 
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curriculum and also obtained high scores on the state test, provided 
evidence that such a combination is not Impossible, at least for a 
high-Income district. On the other hand, the great majority of Items on 
both tests could be successfully Instructed through traditional lecture and 
text methods. 

It is doubtful that even well-designed multiple-choice items can promote 
an activity-based curriculum as effectively as applied performance tests. 
The Eastern State developed optional assessment strategies for teachers 
to accompany Its curriculum framework. They called for students to do 
such things as conduct an experiment or diagram and construct a 
simple circuit while being observed and rated by the teacher. The 
Western State is pro-ceeding to incorporate similar performance items in 
its mandated assess-ment. While such assessments are potentially 
expensive, they appear to be more successful at providing students 
hands-on experience in conducting inquiry than are multiple-choice 
measures. 



State Textbook Adoption Policies 

The Western and Eastern States had policies related to textbook 
adoption. Their use and impact in school districts were affected by four 
features: the alignment of the textbook selection criteria with the state 
curriculum frame-work, the degree to which existing Instructional materials 
met the selection criteria, the resources allocated to the development of 
a state adoption list and the incentives given school districts to select 
instructional materials from the state list. 



Curriculum and Framework Alignment 

The Western State assured alignment between the curriculum frameworks 
and the instructional materials by having the committee members who 
developed the state curriculum framework also develop criteria for 
selecting materials. These criteria were shared with textbook publishers to 
ensure textbooks would be developed to cover the content and intent 
of the state framework. 

The Eastern State's reviewers used general textbook adoption criteria 
supplemented by criteria specific to science. Evaluators completed a 
single-page evaluation summary for each text reviewed, The summary 
required brief comments on strengths, limitations, applications, correlation 
to the state framework and comments, The limited amount of 
evaluative information made it difficult for publishers to understand why 
their books were not put on the state adoption list. 

To complement the general evaluation form, the state science supervisor 
produced more specific criteria. These emphasized fit with the state 
framework and its major goals and philosophy. They also promoted the 
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inclusion of laboratory manuals and activity guides as an btegral part of 
the textbook adoption process, (Since the completion of this study, 
these selection criteria have gained national attention for their utility an d 
thoughtfulness.) 



Resources for Textbook Training 
and Selection 

Frequent criticisms of the process focused on the lack of time allowed 
for selection committees to review as many as 2,000 different texts and 
ancillary materials. The resurt was sometimes the use of the "thumb test; 
where a reviewer quickly fanned through the text or gave a cursory 
glance at the table of contents. In the Western State, the committee 
that developed the state framework also developed the textbook 
selection criteria. Reviewers were trained as a group, in the use of the 
criteria, reviewed the materials independently, then made joint 
recommendations to the state board of education. 

In the Eastern State, a textbook review committee met for two weeks 
during the summer to review instructional materials. Teachers received 
$60 per day to serve on the committee. On the first day, the 
committee was oriented to state science standards and selection criteria. 
The remainder of the time was spent reviewing materials based on the 
generic and science-specific criteria and making recommendations to 
place books on the state adoption list. 



Adoption Incentives 

The Western State encouraged districts to adopt instructional material? 
from the state adoption list by providing districts about $25 per pupil 
annually to belect state-approved textbooks. To assist districts in choosing 
materials, the state published a price list and order form end sent these 
to every county office, district and school. County offices assisted 
schools and districts in ordering textbooks and organizing instructional 
materials display centers. 

The Eastern State had a flexible policy toward textbook adoption. First, it 
provided few resources (about $2 per pupil) to school divisions to 
purchase materials from the state list. Second, the state allowed school 
districts to petition the state board to select books not on the state 
adoption list. Many school districts took advantage of this opportunity 
and were usually success-ful. This set of policies was designed to 
provide state guidance, yet allow school districts some autonomy to 
meet their local needs. 



Implementation Strategies and Technical Assistance 
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Implementation Is the essence of change. Efforts to improve the 
scientific literacy of students, like any efforts at educational improvement, 
will have little impact unless they change what takes place between 
teachers and students in classrooms, it Is a well-known irony, says 
researcher Michael Fullan, that educational innovations have often failed 
because they concentrated on development at the expense of 
Implementation. 

Impacts were highly related to the context of the school district and the 
existence of favorable local conditions. A 19C7 analysis of policy imple- 
mentation by Milbrey McLaughlin suggested that policy success depends 
critically on two broad factors: local capacity and will. "Capacity . . . 
is something that policy can address. Training can be offered. Dollars 
can be provided. Consultants can be engaged to furnish missing 
expertise. But will, or the attitudes, motivation and beliefs that underlie 
an implementor response to a policy's goals or strategies, is less 
amenable to policy intervention." It Is the facet of "will" found in districts 
and schools that is described here. 

If effective change is to result, implementation must not be equated with 
compliance. Districts can "adopt" a state curriculum framework, and 
teachers may even report to principals the objectives they plan to teach 
on given days, without any effective change having taken place, if 
teachers do not understand or accept the premises underlying a 
curriculum, if they lack resources to support it, if they lack the expertise 
to translate it into daily activities, or if they are constrained by such 
things as time, class size or other demands for change, it is unlikely that 
changes in the quality of practice will come chout. 

Districts must have certain factors or conditions resent to implement 
successfully science curricula based on state science frameworks. 

These conditions include: 

• District leadership and commitment to teach science 

• A match between the state framework and district philosophy of 
science or buy-in by the district of the state framework 

• An accountability pressure resulting in a strong desire for a district 
or school to do well on the state assessment 

• Centralization of the science curriculum and instructional materials 
at the district level 

• Teacher involvement in the revision or development of curricula to 
support the state frameworks 

• Available resources for textbooks, hands-on materials or activity kits 
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• The existence of a district-level science supervisor or lead teachers 
In each school 

• Staff development, training and ongoing assistance for teachers 

• District and school-level leadership, monitoring implementation 

State strategies to support these factors helped bring about quality 
implementation in the states studied, 



Strategies to Create a Consensus View 
of Science Education 

Teachers and district administrators will not effectively implement a 
change that they do not understand or that conflicts with their beliefs 
about Instruction. In all three states, state officials addressed this 
dimension at several levels, (a) Influential science educators, including 
local district representatives, were involved in the development of 
curriculum frameworks, (b) Large regional conferences were held to 
Introduce the frameworks and explain them to administrators, (c) State 
science coordinators worked actively to shape local debates on the 
content and purpose of science education. They published articles, 
spoke at local and national conferences of science educators and 
maintained active ties to professional organizations. States were able to 
attract visible and respected experts in science education to leadership 
positions In the state educational agencies, (d) In the Western State, 
the frameworks were incorporated into the curriculum of the regional 
administrator training centers, (e) In the Midwestern State, science 
consultants worked informally to link districts that had strong science 
piograms with nearby districts that were beginning to make changes. 




Strategies for Assisting 
Curriculum Development 

The Western Stcite developed lesson strategies to expand upon the 
science curriculum framework grid provide teochers options in teoching 
specific objectives. All three stotes encouraged locol districts to develop 
their own expanded curriculum guides based upon state frameworks. 
They employed several techniques to assist in this process, (a) The 
Western Stcite formalized a three-year adoption process tied both to 
textbook adoption and to the cycle for revising state frameworks. Local 
educators were familiar with the cycle and used it. (b) States exerted 
Influence on the use of federal funds from Title II of the Education and 
Economic Security Act (EESA). This act, administered by the states, 
provides grants to local and intermediate districts to improve science and 
mathematics instruction. States specified that the funds be used for 
release time to support curriculum or staff development to implement 
state science or math frameworks. 

28 



Creation of District Science 
Specialist Positions 

For small districts, the absence of an individual with primary responsibility 
for Implementing the science program was a significant handicap. 
Likewise, the designation of a lead science teacher in elementary schools 
contributed appreciably to the Implementation of a quality science 
program. In one district, Title II funds were used to purchase release 
time for one teacher at each elementary school one day a week to 
serve this function. 



Strategies for Assisting 
with Teacher Training 

Each state provided some teacher training directly through the state 
educa-tlon department, but none of these efforts was well funded. The 
Western State used regional centers to promote the implementation 
cycle, but these were later disbanded because of funding reductions. In 
the Eastern State, state science consultants offered occasional courses 
and regional workshops within a travel budget of less than $2,600 per 
person. In the Midwestern State, the one state science consultant 
provided on-site technical assistance to a few districts upon request, 
within extreme limitations on time and budget. 

Given these restraints, districts relied heavily on university professors and 
their own expertise for teacher training to support the implementation of 
new science frameworks. As pointed out previously, the activity and 
science-process focus of all three state frameworks presented a significant 
change for many districts and a clear need for training. At least one 
state made an active if informal effort to identify university and district 
consultants whose view of science Instruction corresponded with thai- 
expressed in the state frameworks and to form a network of trainers. 

The states also used EESA Title II money for local training to support the 
implementation of the curriculum frameworks. 



Strategies for Assisting 

With Monitoring and Evaluation 

Each state used a different approach to monitor the implementation of 
science education improvements. In the Eastern State, the one state 
that required use of the state framework "or equivalent," school 
accreditation teams checked for documentation, usually in the form of 
skeletal lesson plans, that prescribed objectives were being taught. While 
this practice could not establish the quality of implementation, and could 
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potentially Interfere with It, it did maintain awareness of the state 
framework and was generally supported by the teachers Interviewed. 

To examine program quality In greater depth, the Eastern State 
developed a science education program assessment Inventory, a net of 
extensive questionnaires for parents, teachers, students and administrators. 
These were to be used In conjunction with an on-site visit by a team of 
science educators from other districts. The approach had not been 
Implemented at the time of the study. 

In the Western State, program review was coordinated with the state's 
school Improvement program. One step of this process Involved an on- 
site program review by a visiting team. District review teams were made 
up of educators from outside a district who had been trained and 
certified by the state; training focused on implementation of the state 
curriculum frameworks. In addition, the Improvement process inchded an 
internal review focusing on subject areas In a cycle coordinated with 
that of textbook adoption and curriculum revision. 



JMPLICATIONS FOR STATE POLICY. 



The evidence examined In this study, on balance, supports state efforts 
to Improve science education In the elementary grades (K-8) through the 
use of curriculum frameworks, student assessment and textbook selection 
criteria. Analysts Linda McNeil and Arthur Wise recently have raised 
concerns that such initiatives undermine the professionalism of teachers 
and result In superficial texts and fragmented or narrowly focused 
Instruction. Such concerns should be examined seriously, but they were 
not supported by this survey and case studies of science education in 
three diverse states. 

State science curriculum frameworks, varied as they were, were valued 
by teachers and administrators because they fulfilled a need to define 
the domain and goals of science education. Mandated assessments 
acknowledged the Importance of science instruction, contributed to new 
efforts In some districts to strengthen science Instruction, and reinforced 
the new focus of curriculum frameworks on the social consequences of 
technology and the process of scientific Investigation. 

While it was difficult to assess the impacts of textbook selection criteria 
directly, district curriculum experts perceived improvements in the 
materials available to them, and supported the practice of first defining 
what should be taught and then selecting materials accordingly. The 
emphasis In state curriculum frameworks on Investigative activities was 
reflected in the selection of texts providing support for such activities, 
Including software and equipment. 

The positive nature of these findings may reflect the status of elementary 
science education In ways that may not generalize to other areas of the 
curriculum. Elementary teachers are In general less well trained in 
science and less comfortable teaching science than, say, reading or 
social studies. Student achievement In science Is not assessed In most 
districts apart from state assessments. Many districts do not have the 
resources and expertise to develop science curricula apart from 
textbooks, and consequently can provide little support for activity-based 
Instruction. Moreover, earlier state policies emphasizing the Importance of 
reading and mathematics have contributed to a de-emphasis of science 
until very recently. For such reasons, it is quite possible that state 
curriculum frameworks and assessment activities could have a more 
positive and substantial effect in science than in some other subject area 
because they address a more substantial need. 

The impact of state efforts to improve instruction and curriculum are 
greatest when curriculum frameworks, student assessment and textbook 
criteria are approached together as an integrated strategy, with the 
curriculum frame-wc.k at the core. Case studies indicated that the 
tailoring of assessment to reflect the curriculum framework was important 
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to district decisions to change, Integration of this sort has organizational 
implications for departments of education. General recommendations 
are: 

• The development of curriculum frameworks should precede the 
development of assessment instruments or selection criteria for 
materials. Cycles of revision should also begin with curriculum. 

• Direct and regular communication between curriculum units and 
assessment units Is essential. Consideration should be given to 
housing these units close to each other and placing them under a 
single upper-level administrator. 

• Some subject-matter experts Involved in the development of 
curriculum frameworks should also play a direct role In the 
development of assessments and of textbook selection criteria. 



Curriculum Frameworks 

Two primary local purposes were served by the curriculum frameworks in 
the states studied. They provided an optional definition to local districts 
of what science education ought to be which often extended well 
beyond what districts had Implemented in classrooms previously. In this 
respect, they served an educational function in themselves. They also 
facilitated district decision-making regarding the content of staff training, 
the purchasing of materials and equipment, program review and the like. 



At the state level, they served as the basis for other state actions. 
Taken together, these purposes suggest that the quality of the science 
framework is particularly important. Several suggestions flow from the 
experience of the states and districts in this study. 

• To Insure an acceptable level of quality, the framework should 
reflect the thinking of leading national organizations such as the 
National Science Foundation, the American Association for the 
Advancement of Science and the National Science Teachers 
Association. As these organizations become more involved In goal 
setting for science education, these trends should be incorporated 
into state curriculum frameworks. 

• The participation of Influential districts In the development of 
frameworks is Important for buy-in, but quality should not be 
compromised to achieve broader participation. The Eastern State 
was not able to achieve greater acceptance of its framework than 
the Western State, yet its objectives were authored by 96 
individuals from across the state, compared to the Western State's 
committee of 14. 
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• Objectives must be broad enough to give direction to Instruction, 
but not so specific that Instruction becomes atomistic. For example, 
one objective that students "use data to reach conclusions" was 
Identified by some teachers as too broad to guide either Instruction 
or assessment. 

• Adherence to frameworks should be optional. Some districts had 
curriculum frameworks that were significantly different from the 
state's. Some of these districts had well-conceived and well- 
Implemented pro-grams In place, reflecting years of staff 
development and large investments in support structures and 
materials. To demand the restructuring of such a program 
because it does not correspond to a state framework implies that 
there Is only one correct approach to science curriculum, an Idea 
foreign to the tentative nature of science itself. 



Assessment 

Assessment was an Important element In each district that underwent 
significant change In science instruction following state actions. In these 
districts, assessment or the anticipation of assessment brought attention 
and resources to science because district administrators wanted to avoid 
the embarrassment of having low scores publicly reported. When tests 
were ongoing, they served as an annual scorecard, an indication of 
whether a school or district was doing better or worse. In this respect, 
they were also a formal, if tacit, indication of what really mattered to a 
district. When science was tested, It mattered more than when It was 
not tested. In subjects that had been tested for years and clearly 
mattered, such as mathematics, test scores were frequently analyzed to 
identify particular skills needing greater emphasis or revised teaching 
strategies. 

• Tests shall explicitly measure the attainment of state science 
objectives. In the Eastern State, where the assessment was not 
perceived to match state objectives, teachers ignored the results of 
the science test. In the Western State, where test and objectives 
were aligned, teachers were more inclined to view the test as an 
indication of how well students were learning science. 

• School-level reporting should be used to bring an appropriate level 
of accountability pressure. While district administrators and 
principals often expressed resentment regarding school-level 
reporting of test results, it appeared that the consequences of 
these pressures were positive. Sensitivity to test publicity motivated 
principals and central administrators to attend to test results and to 
bring them to the attention of teachers. In the Western State, 
reporting scores within ranges achieved by schools of similar 
socioeconomic status appropriately prevented schools serving low- 
Income students from experiencing undue pressures. At the same 
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time, the fact that student -and classroom scores were not 
published and that decisions about individual students were not 
made on the basis of test scores Insulated teachers from pressures 
to raise scores at the expense of sound educational practice. 

• To avoid narrow teaching to particular test Items, a known set of 
test Items should not be repeated year after year. In the 

Midwestern and Eastern States, teachers Indicated high familiarity 
with state test items In language arts because the scores were 
Important and the same items were repeated over a period of 
years. The possibility existed that Improved scores came about 
because teachers taught narrowly to the topics tested. The 
Western State avoided this problem by sampling test items, creating 
35 forms of the 8th-grade science test. 

• Performance testing should be considered to promote activity-based 
instruction. In the Midwestern State, test developers made a 
concerted effort to develop muitiple-cholce Items to measure 
science process skills that would be perceived by teachers as 
encouraging hands-on Instruction. Teachers were not sufficiently 
familiar with the Items for researchers to evaluate them effectively. 
The evidence was neutral In balance: while some teachers 
believed that students would perform better on the test If they had 
engaged in activities, some also believed that activities were a 
more time-consuming way to com-munlcate Information. Direct 
assessments of students' performance In conducting an observation 
or testing a hypothesis, while considerably more expensive, would 
have unambiguous Implications for Instruction. 



Textbook Selection 

The effect of the "dumbing down" of textbooks had a more negative 
impact on the Western State than the Eastern State. Because the 
Western State's science framework was Innovative and state-of-the-art, 
existing teAtbooks (based on consensus objectives from other states) did 
not reflect the state's new framework or model curriculum guide. 
Pressure was placed on publishers to revise the texts to match the state 
framework. Although the state role in the textbook Issue is thorny and 
complex, this study supports the following conclusions: 

• States and national organizations should work together to develop a 
curriculum "core" In science which becomes the basis of content 
Included In textbooks. By defining a common core of content, 
textbooks can move away from "mentioning" topics and treating 
them superficially. Fower concepts and topics could be covered In 
greater depth, aiding student understanding and learning of 
science. 
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• Textbooks should reflect the current goals of science, Including 
applications of science to everyday life; science, technology and 
society; the mature and process of science and ethical Issues. 

Although there is no national or professional consensus on what the 
goals of science should be, texts still reflect a content and 
knowledge orientation to presenting science to students, 

• Textbook adoption criteria should reflect the state curriculum 
framework. Criteria should facilitate the selection of high-quality 
textbooks to support the curriculum. Although correlational analysis 
will indicate the framework content covered by a text, more 
Important is the approach, writing and presentation of the subject 
matter. Weight should be given to the book's overall academic 
integrity, its clarity, quality of writing, factual accuracy, its interest to 
students and capacity to motivate and stimulate them. If a 
"hands-on" approach to teaching science is desired, then 
supplemental materials like activity kits and the orientation of the 
textbook to "hands-on" learning should be part of the selection 
criteria. 

• Sufficient resources should be dedicated to the selection process. 

Reviewing existing textbook series is a massive task. The selection 
committee should be paid and allowed enough time to be 
appropriately trained to do a thoughtful review of all instructional 
materials considered. 

• Financial Incentives should be provided to encourage districts to 
purchase textbooks on the state adoption list, thus fostering 
curriculum alignment with the state framework. The Western State 
encouraged districts to select books on the state adoption list by 
reimbursing them for most of the per-pupil cost of textbooks. 
Because this state selected a limited list of acceptable textbooks, 
congruence with the state framework was virtually guaranteed. 
However, In the Eastern State, where alignment of the curriculum, 
assessment and textbook list was purposely loose, it seemed 
appropriate to allow districts to select books not on the state 
adoption list to provide some local control and discretion. 



Dissemination, Implementation 
and Monitoring 

The use of comprehensive dissemination strategies greatly influenced the 
use of state frameworks in districts. The first task was to get the 
frameworks into the hands of local educators. The second task was to 
get educators to buy into and understand the philosophy, approach and 
content of the framework to facilitate their use and implementation. 

• Dissemination of the state science framework should be 

accomplished through the use of multiple, comprehensive strategies. 
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These could include regional conferences sponsored by the state 
education agency, the mailing of the state frameworks to every 
school in the state, media reports, dissemination through state 
professional assocla-tlons like the state affiliates for NSTA and the 
involvement of local educators in the development process. 

• States should provide both pressure and support to school districts 
to facilitate Implementation. Pressure can come in the form of an 
accountability pressure, public reporting of test scores, accreditation 
requirements or school improvement evaluation criteria. Support 
can be discretionary resources or technical assistance. 

• Discretionary resources should be provided to give districts flexibility 
during implementation. Resources, in the form of Title II funds, 
school improvement funds, teacher training funds or other 
discretionary funds are necessary for districts to buy instructional 
materials, release time for teachers, outside expertise or provide 
additional stipends for lead teachers. 

• Technical assistance, in the form of teacher training, curriculum 
Implementation or the development of mentor or lead teachers 
should be made available to school districts. Most districts have 
limited resources in terms of science curriculum expertise and staff 
development opportunities. To the extent the state can provide 
technical assistance to build capacity in a district, implementation is 
facilitated. 

• State-sponsored regional centers should provide ongoing assistance 
to districts to promote curriculum implementation. In the Western 
State, regional centers played a major role in helping districts 
implement the three-year cycle of curriculum implementation. The 
availability of assistance on an ongoing basis is critical to fidelity 
and completeness of implementation. 

• Monitoring, In the form of regulations or Incentives, should be 
carried out to promote Implementation. In the Eastern State, tying 
the teaching of the learner outcomes to the state accreditation 
requirements forced the teaching of the state framework. In the 
Western State, program review criteria for a school improvement 
program (which had financial incentives) were keyed to the 
implementation of the state curriculum frameworks. 

In short, science curriculum improvement occurs when there is a careful 
articulation of state policies related to assessment, curriculum, instructional 
materials and school improvement and when there is both pressure and 
support to implement the policies. 



26 



36 



NOTES 



Michael Fullan. The Meaning of Educational Change . New York: 
Teachers 

College Press, 1982. 

Linda McNeil. Contradictions of Control: School Structure and School 
Knowledge . New York: Methuen/Routledge & Kegon Paul, 1986. 

Mllbrey McLcughlln. "Learning from experience: Lessons from policy 

Implementation," Educational Evaluation and Policy Analysis . Summer 
1987, pp. 171-178. 

Arthur Wise. "Legislated learning," Phi Delta Kappan , January 1988, 
pp. 328-333. 



9 

ERJ.C 



37 

27 



